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Problem Context ( 1 ) 



• Ongoing maintenance by the sustaining 
engineering group has limited personnel and 

test resources but must maintain 
99.9% proficiency and 97.0% 
availability of systems 

• Major new capabilities fielded for the Space 
Network are often contracted to entities 
outside of the sustaining engineering group 
and historically have higher than required 
defect rates 
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Problem Context (2) 



• Software has historically accounted for an 
annual average of 28% of the Space Network 
loss of availability and proficiency (low of 
11% and high of 57% annually) 



• CSCI A and CSCI B account for 42% of the 
previous eight months software data loss 


Percentages reflect the portion of losses attributed to software with respect to the total loss within the control of the 
Space Network. For example, if the annual average loss attributable to the Space Network was 500 out of 130k 
total hours (99.996% proficiency), then the portion caused by software would be 500 x 0.28 = 140 hours. 
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Problem Context (3) 
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33 

7% 
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9.748 
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16.784 
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0.002 


• CSCI A and CSCI B have discrepancies an 
order of magnitude larger than other CSCIs 

• CSCI A was deployed several years ago 

• CSCI B was deployed eight months ago 

• These data represent the past eight months 
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Problem Context (4) 



• CSCI A represented approximately 15% of all 
scheduled support time over the past eight 
months but accounted for 37% of all loss 

• CSCI B has not had schedule data long 

enough to trend schedule time versus loss 
^ had delayed deployment due to problems 
found during system and acceptance testing 

• The remainder of the Space Network systems 
accounted for 58% of the data loss while 
providing at least 75% of scheduled support 
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Analysis Results (1) 



• Running CodeSonar and performing 
preliminary review of the results averaged 3.5 
minutes per finding (approx. 20 hours total) 

• An additional 40 hours is estimated 

to analyze the 37 findings deemed too 
complex for the initial review 

• Using CodeSonar’s tools to suppress 

known non-problems, delta tool runs will not 
repeat findings marked as non-problems, 
further reducing the time needed for review 
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Analysis Results (2) 



• Of the 330 total findings, 70 
were associated with the 
Standard Template Library 
(STL) and 109 had duplicate 
sources to other findings 
(fixing one either fixes the 
duplicate problems or provides 
a straightforward method to fix 
the duplicate problems) 
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Analysis Results (3) 
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Of the remaining 151 
findings, 29 are findings that 
could potentially be problems 
but the coding style and 
standards used prevent the 
problem 

An addition 29 are definite 
false positives, which were 
determined to not be 
problems upon review 
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Analysis Results (4) 



Of the final 93 findings, 37 
require more review because o 
the complexity of the code in 
where the potential defect 
exists, 5 are considered urgent 
(will cause data corruption or 
processor reset) and 37 are 
considered routine (worth 
correcting as time is available) 
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Findings Comparisons (1) 




Data has just recently 
started being collected 
on the amount of time 
required to investigate 
DRs and provide a 
complete fix, so no 
trends can yet be 
provided regarding 
time and dollars 
expended on defects 
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Findings Comparisons (2) 



• The average data loss per software discrepancy 
is from 2003 to 2007 was 0.331 hours 

• The average data loss per software discrepancy 
over the past eight months is 0.579 which may 
be attributable to additional users of CSCI A in 
addition to fielding CSCI B 

• A conservative estimate of savings of data loss 
by correcting the 5 urgent findings is 2.895 
hours of data loss, which is 1 1.5% of the 
estimated annual software data loss for 2008 
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Findings Comparisons (3) 



• Jet Propulsion Laboratory (JPL) performed a 
case study that found 1 .2 fielded defects / 
LKSLOC in ground software systems they 
developed 1 

• Capers Jones found that CMMI Level 5 
organizations deliver 1 .05 defects / KLOC 2 

• Of the 12 CSCs analyzed in this study, all fall 
below both these defect rates except two with 
respect to the true positive findings 


Credits: 


1 - Spagnuolo, J. and Powell ,J. Defect Measurement and Analysis of JPL Ground Software: A Case Study. 2004. 

2 - Jones, C. Software Assessments, Benchmarks, and Best Practices. 2000. 
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Findings Comparisons (4) 



• The Coverity project to use static analysis 
technology on Open Source software produced 
an average of 0.605 findings / KSLOC over 96 
projects using a limited set of the most easy to 
understand defects 

• CodeSonar produced an average of 0.730 
findings that were confirmed to be valid, 
which implies that improvements should be 
made in the overall quality of CSCI A and 
CSCIB 
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Acceptance Measures (1) 



Measures are meant to aid in determining the 
impact of the technology on the proj ect both in the 
expenditure of effort and the technical results of the 
technology 


Findinq CIs 

ssification and 

Prioritization Counts 


Urgent 

Routine 

All Prioritizations 

True Positive 

1 

2 

3 
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Recurrina 

Staff Hours 

Run Time 

0.5 

Result Review 

2 



KSLOC Analyzed 
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Average data loss per defect 


Average time to correct defect 
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Acceptance Measures (2) 







Time will be added to sustaining 
engineers’ lifecycle to run CodeSonar 
and disposition the findings 

After relatively small initial investment 
for the initial tool run (compared to 
anecdotal comparisons to discrepancy 
investigation time), an estimated 10 
minutes per KSLOC developed of 
extra time is anticipated based on 
preliminary use of the tool (without 
formal training) 
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Acceptance Measures (3) 




The “non-interesting” finding rate of 
70% is a large number but filtering, 
search, and detailed contextual features 
of CodeSonar reduce the time per 
finding 

Integration of tool into the build 
process may also provide further 
savings by preventing developers from 
having to configure and operate the 
tool separately 
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Conclusions (1) 



• Preliminary results show the tool to be easy to 
use and incorporate into engineering processes 

• Preliminary findings give significant potential 
improvements in proficiency and availability 
on the part of software 
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Conclusions (2) 


• As time-to-fix data becomes available a better 
cost trade can be made on person hours saved 
versus tool cost 

• Selective factors may be necessary to 
determine where best to apply CodeSonar to 
balance cost and benefits 
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